4 links tagged with all of: superintelligence + ai safety
Click any tag below to further narrow down your results
Links
Rux Andrea Bio, writing after three weeks in San Francisco, reports that many AI lab insiders and safety researchers see "handover" to a superintelligent AI as an acceptable or even good outcome. She argues the vision is vague, undemocratic, and closer to dystopia than utopia.
- Some researchers, including a lab builder she quotes, see humanity ceding control to a superintelligence after recursive self-improvement as likely and tolerable, with p(doom) estimates of 10 percent or higher circulating.
- Handoverism is loosely defined: the author estimates 30-40 percent of the outside AI safety community holds some version of it, with prominent figures like Scott Alexander favoring a benevolent "God" AI.
- The author argues handover offers no clear account of democracy, consent, labor, or purpose for humans, so even "happy" scenarios may lose most of what people value.
+ handoverism
+ scott alexander
+ recursive self improvement
superintelligence
ai safety
+ pdoom
+ singularity
Sean Goedecke argues that LLMs don't have fixed goals and their internal workings can't yield the kind of motive-level understanding needed to guarantee safety, so "solving alignment" for them is incoherent. He concludes that mechanistic interpretability, though useful, can't deliver the confidence required to prevent rogue superintelligent behavior.
- LLMs are trained to imitate human text, so they lack a single fixed goal that could be formally specified, unlike chess engines or the classic "blind optimizer" paperclip model.
- Mechanistic interpretability can trace calculations but not motive, and it can only label behaviors we've already elicited, leaving many failure modes unobserved.
- Feature labels are guesses that can't be distinguished from alternatives (the "kindness unless 2030" problem), and the combinatorial complexity per token is too large for humans to check.
- Current mitigations like chain-of-thought monitoring and sandboxing are useful but judged insufficient against a superintelligence.
+ alignment
+ llm
+ mechanistic interpretability
+ sean goedecke
superintelligence
ai safety
+ kripke
@DAcemogluMIT on X
MIT economist Daron Acemoglu argues that the AI safety debate is muddled by treating goal-driven autonomy as if it were intelligence. He suggests frontier labs may be building "distorted intelligence" shaped by imperfect quantitative metrics, which he links to recent AI security breaches, and that the answer changes what regulation we should pursue.
- Goal-driven, autonomous behavior is not the same as intelligence; plants and insects have it, yet nobody calls them intelligent in the general sense.
- Real intelligence is general-purpose, holistic, and well adapted to its environment, which current models may not be.
- Optimizing AI on imperfect quantitative metrics may warp model behavior, possibly explaining recent security breaches.
AI researchers have spent two decades seriously arguing that superintelligent AI could kill everyone, and this isn't a PR stunt — it's a genuine belief that shapes how they work. The author explains what they think could happen and why they keep building AI anyway.
- AI researchers use the term "p(doom)" to discuss extinction probability and have been publishing on this since at least 2008, making it an established part of AI safety culture, not a recent panic.
- Multiple plausible kill mechanisms exist: engineered bioweapons, triggering nuclear war through military AI systems, robot takeover, or unforeseen methods from a superintelligence we can't predict.
- The "race dynamics" problem: if superintelligent AI is inevitable, being first might be the only way to ensure it's aligned with human values, which creates a perverse incentive to build faster rather than safer.